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EXECUTIVE  SUMMARY 


OBJECTIVES  OF  THE  DEMONSTRATION 

In  2003,  the  Defense  Science  Board  observed:  “The  problem  is  that  instruments  that  can  detect  the 
buried  unexploded  ordnance  (UXO)  also  detect  numerous  scrap  metal  objects  and  other  artifacts, 
which  leads  to  an  enormous  amount  of  expensive  digging.  Typically  100  holes  may  be  dug  before 
a  real  UXO  is  unearthed!  The  Task  Force  assessment  is  that  much  of  this  wasteful  digging  can  be 
eliminated  by  the  use  of  more  advanced  technology  instruments  that  exploit  modern  digital 
processing  and  advanced  multi-mode  sensors  to  achieve  an  improved  level  of  discrimination  of 
scrap  from  UXO.”  Since  2003,  significant  progress  has  been  made  in  UXO  classification 
technology.  To  date,  testing  of  these  approaches  has  been  primarily  limited  to  test  sites  with  only 
limited  application  at  live  sites.  Acceptance  of  classification  technologies  requires  demonstration 
of  system  capabilities  at  real  UXO  sites  under  real  world  conditions.  Any  attempt  to  declare 
detected  anomalies  to  be  harmless  and  requiring  no  further  investigation  will  require  demonstration 
to  regulators  of  not  only  individual  technologies,  but  an  entire  decision  making  process. 

At  the  Spencer  Range  Tennessee  site  covered  by  this  report,  the  objective  was  to  discriminate 
targets  of  interest  (TOI)  (including  37-millimeter  [mm],  60-mm,  75-mm,  105-mm,  155-mm 
targets  and  small  and  medium  industry  standard  objects  [ISOs])  from  non-hazardous  shrapnel, 
range  and  cultural  debris.  We  describe  the  performance  of  classification  techniques  that  utilized: 
(1)  full  coverage,  dynamically  acquired  survey  data  collected  with  both  the  Time  Domain 
Electromagnetic  Towed  Array  Detection  System  (TEMTADS)  2x2  and  the  MetalMapper 
advanced  electromagnetic  induction  (EMI)  sensors;  and  (2)  static,  cued  interrogation  style  data 
acquired  with  MetalMapper  and  TEMTADS  2x2. 

TECHNOLOGY  DESCRIPTION 

The  Advanced  Geophysical  Classification  (AGC)  techniques  applied  to  the  Spencer  Range  data 
use  dipole  model-based  features  extracted  from  multi-static,  multi-channel  EMI  data.  Single 
source  and  multi-source  dipole  inversions  were  used  to  estimate  target  location,  orientation,  and 
principal  polarizabilities.  From  the  extracted  feature  vectors,  prioritized  dig-lists  were  created 
for:  (1)  the  MetalMapper  deployed  in  a  dynamic,  full  coverage  mode;  (2)  the  MetalMapper 
deployed  in  a  static,  cued  mode  (for  2  unique  datasets  over  identical  targets  collected  by  Naeva 
Geophysics  Inc.  [NAEVA]  and  URS  Corporation  [URS]);  (3)  the  TEMTADS  2x2  deployed  in  a 
dynamic,  full  coverage  mode;  and  (4)  TEMTADS  2x2  deployed  in  a  static,  cued  mode, 

Anomalies  were  prioritized  based  on  a  match  of  estimated  principal  polarizabilities  to  a 
polarizability  library  of  known  TOIs,  polarizability  magnitude,  and  the  rate  of  decay  of 
polarizabilities.  For  each  dataset,  a  reference  library  of  polarizabilities  was  constructed  from  test  pit 
measurements,  site  specific  training  data,  and  polarizabilities  derived  from  data  acquired  at 
previous  Environmental  Security  Technology  Certification  Program  (ESTCP)  demonstration  sites. 
Since  dynamic  data  have  higher  noise  levels  than  cued  data,  a  more  conservative  classification 
algorithm  -  the  Combined  Classifier  Ranking  (CCR)  algorithm  -  was  used.  The  MetalMapper 
(NAEVA)  cued  data  were  processed  automated  classification  that  combines  multiple  ranking  rules 
(e.g.  size,  decay,  etc.)  and  library  matching.  Classification  parameters  were  determined  by 
DigZilla,  with  the  only  input  by  the  analyst  being  the  reference  library  of  ordnance  polarizabilities. 
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All  model  fits  and  classification  analysis  were  performed  using  a  classification  software  suite 
(UXOLab)  that  was  jointly  developed  by  the  University  of  British  Columbia  -  Geophysical 
Inversion  Facility  and  Black  Tusk  Geophysics  (BTG). 

DEMONSTRATION  RESULTS 

Performance  metrics  defined  by  the  ESTCP  program  office  were  calculated  for  each  data  set. 
Application  of  dipole  based  classification  was  successful  for  all  data  sets,  with  100%  of  TOI 
being  identified  and  marked  for  excavation  in  each  case.  The  reduction  of  clutter  digs  while 
retaining  all  TOI  was  greater  than  80%  in  all  cases.  Not  surprisingly,  the  data  sets  processed  with 
the  more  conservative  CCR  algorithm  had  a  higher  percentage  of  false  alarms  than  the  data  sets 
processed  with  a  more  aggressive  approach.  The  number  of  "can't  analyze"  anomalies  met  the 
success  criteria  for  all  data  sets  except  the  MetalMapper  (URS)  dynamic  data  acquired  in  the 
Dynamic  area.  For  the  dynamically  acquired  data,  reliable  target  parameters  were  estimated  for 
only  92.1%  due  to  the  lack  of  data  coverage  at  the  edges  of  the  survey  area. 

For  the  cued  data  sets  with  inertial  measurement  unit  (IMU)  information  and  the  dynamically 
acquired  data,  the  target  location  estimate  error  had  a  standard  deviation  of  less  than  10 
centimeters  (cm).  Data  acquired  in  the  Open  area  and  Dynamic  area  had  depth  estimate  errors 
with  a  standard  deviation  of  less  than  10  cm.  The  data  acquired  in  the  Treed  area  by  the 
TEMTADS  2x2  had  a  depth  estimate  error  standard  deviation  of  13  cm,  which  did  not  meet  the 
success  criteria  of  having  a  depth  estimate  error  with  a  standard  deviation  of  less  than  10  cm. 
The  survey  conditions  in  the  Treed  area  may  have  resulted  in  variation  of  the  ground  clearance 
height  of  the  instrument,  whereas  we  assumed  a  fixed  ground  clearance  height  for  all  anomalies. 

IMPLEMENTATION  ISSUES 

A  major  implementation  issue  with  AGC  technology  is  providing  easy  to  use  software  tools  so 
that  non-expert  staff  can  obtain  good  classification  performance.  This  project  included  a 
technology  transfer  and  training  component.  A  member  of  Shaw  Environmental’s  production 
team  attended  a  one  week  training  session  in  Vancouver,  British  Columbia,  Canada  with  BTG 
algorithm  and  software  developers.  The  training  session  included  an  overview  of  UXO  inversion 
and  classification  theory  and  software  routines.  The  Shaw  geophysicist  was  responsible  for 
executing  all  parts  of  the  classification  workflow:  from  data  and  inversion  quality  control  (QC), 
training  data  selection,  to  dig  list  creation  and  submittal.  The  dig  list  submitted  by  the  Shaw 
geophysicist  successfully  identified  all  TOI  and  greatly  reduced  the  number  of  non-TOI  digs. 
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1.0 


INTRODUCTION 


1.1  BACKGROUND 

In  2003,  the  Defense  Science  Board  observed:  “The  problem  is  that  instruments  that  can  detect  the 
buried  unexploded  ordnance  (UXO)  also  detect  numerous  scrap  metal  objects  and  other  artifacts, 
which  leads  to  an  enormous  amount  of  expensive  digging.  Typically  100  holes  may  be  dug  before 
a  real  UXO  is  unearthed!  The  Task  Force  assessment  is  that  much  of  this  wasteful  digging  can  be 
eliminated  by  the  use  of  more  advanced  technology  instruments  that  exploit  modem  digital 
processing  and  advanced  multi-mode  sensors  to  achieve  an  improved  level  of  discrimination  of 
scrap  from  UXO.”  Since  2003,  significant  progress  has  been  made  in  UXO  classification 
technology.  To  date,  testing  of  these  approaches  has  been  primarily  limited  to  test  sites  with  only 
limited  application  at  live  sites.  Acceptance  of  classification  technologies  requires  demonstration 
of  system  capabilities  at  real  UXO  sites  under  real  world  conditions.  Any  attempt  to  declare 
detected  anomalies  to  be  harmless  and  requiring  no  further  investigation  will  require  demonstration 
to  regulators  of  not  only  individual  technologies,  but  an  entire  decision  making  process. 

To  demonstrate  the  viability  of  advanced  detection  and  discrimination  technologies,  the 
Environmental  Security  Technology  Certification  Program  (ESTCP)  has  now  conducted  multiple 
UXO  classification  studies.  The  results  of  the  first  demonstration,  at  the  former  Camp  Sibert, 
Alabama  were  very  encouraging.  Although  conditions  were  favorable  at  this  site,  including  a 
single  target-of-interest  (4.2-inch  [in]  mortar)  and  benign  topography  and  geology,  all  of  the 
demonstrated  classification  approaches  were  able  to  correctly  identify  a  sizable  fraction  of  the 
anomalies  as  arising  from  non-hazardous  items  that  could  be  safely  left  in  the  ground.  Of 
particular  note,  the  contractor  EM-61-MK2  cart  survey  with  analysis  using  commercially 
available  methods  correctly  identified  more  than  half  the  targets  as  non-hazardous. 

To  build  upon  the  success  of  this  first  study,  ESTCP  expanded  the  program  to  include  a  second 
study  at  a  site  with  more  challenging  topography  and  a  wider  mix  of  targets  of  interest  (TOIs).  A 
range  at  the  former  Camp  San  Luis  Obispo,  California,  was  selected  for  this  demonstration.  We 
again  found  that,  with  appropriate  use  of  classification  metrics  applied  to  production  quality  EM- 
61  data,  it  was  possible  to  significantly  reduce  the  number  of  clutter  items  excavated  without 
missing  any  TOI.  Furthermore,  the  next  generation  of  electromagnetic  induction  (EMI)  sensors, 
when  deployed  in  a  cued-interrogation  mode,  produced  significant  additional  reductions  in  the 
number  of  clutter  items  excavated.  These  sensors  could  also  usually  distinguish  between 
different  UXO  types.  A  third  ESTCP  demonstration  study  was  conducted  in  2010  at  Camp 
Butner,  North  Carolina.  The  site  had  very  little  topographic  relief  but  required  classification 
between  small  TOIs  (37  millimeter  [mm]  projectiles  and  M48  fuzes)  and  metallic  debris  of 
similar  size.  Targets  were  also  distributed  with  a  higher  density  than  previously  encountered.  In 
2011,  an  ESTCP  demonstration  study  was  conducted  at  Camp  Beale,  California.  The  site  had 
very  little  topographic  relief  but  required  classification  between  small  TOIs  (37-mm  projectiles 
and  M48  fuzes)  and  metallic  debris  of  similar  size.  Targets  were  also  distributed  with  a  higher 
density  than  previously  encountered.  Also  in  2011,  a  study  was  conducted  at  Pole  Mountain, 
Wyoming.  The  smallest  TOI  at  this  site  was  a  small  industry  standard  object  (ISO).  Library 
based  classification  applied  to  features  derived  from  cued  MetalMapper  data  resulted  in  some  of 
the  best  classification  performances  to  date,  with  less  than  5%  of  non-TOI  requiring  excavation 
before  all  TOI  were  recovered. 
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1.2  OBJECTIVES  OF  THE  DEMONSTRATION 


The  objectives  of  this  demonstration  were  to  perform  data  modeling,  classification,  and 
classification  using  electromagnetic  (EM)  data  collected  by  the  various  data  collection 
demonstrators  participating  in  the  study.  Specifically,  we  processed  the  following  datasets 
collected  at  Spencer  Range: 

1)  Time-domain  Electromagnetic  Towed  Array  Detection  System  (TEMTADS)  2x2 
dynamic  data; 

2)  TEMTADS  2x2  cued  interrogation  data; 

3)  MetalMapper  dynamic  data; 

4)  MetalMapper  cued  interrogation  data  acquired  by  URS  Corporation  (URS);  and, 

5)  MetalMapper  cued  interrogation  data  acquired  by  Naeva  Geophysics  Inc.  (NAEVA). 

The  Spencer  Range  demonstration  site  was  divided  up  into  three  distinct  areas  differentiated  by 
the  sensors  deployed:  “Open”  area;  “Treed”  area;  and  “Dynamic”  area.  The  Open  area  was 
surveyed  in  a  cued,  static  mode  by  the  MetalMapper.  Cued  MetalMapper  data  were  acquired  by 
NAEVA  and  URS  using  the  same  anomaly  list.  A  more  challenging,  treed  survey  area  was 
surveyed  by  cued  portable  sensors  including  the  HandHeld  Berkeley  UXO  Discriminator  (BUD), 
Man  Portable  Vector  (MPV)  and  TEMTADS  2x2.  There  was  also  a  relatively  small  dynamic 
area  that  was  surveyed  by  the  TEMTADS  2x2,  MPV  and  MetalMapper  in  both  cued  and 
dynamic  modes  to  provide  cued  versus  dynamic  deployment  comparison  for  identical  targets. 

All  performance  metrics  except  two  were  met.  For  the  MetalMapper  (URS)  dynamically 
acquired  data,  reliable  target  parameters  were  estimated  for  only  92.1%  due  to  the  lack  of  data 
coverage  at  the  edges  of  the  survey  area.  The  data  acquired  in  the  treed  area  by  the  TEMTADS 
2x2  had  a  depth  estimate  error  standard  deviation  of  13  centimeters  (cm)  due  to  variations  in  the 
ground  clearance  height. 

1.3  REGULATORY  DRIVERS 

The  Defense  Science  Board  Task  Force  on  UXO  noted  in  its  2003  report  that  75%  of  the  total 
cost  of  a  current  clearance  is  spent  on  digging  scrap.  A  reduction  in  the  number  of  scrap  items 
dug  per  UXO  item  from  100  to  10  could  reduce  total  clearance  costs  by  as  much  as  two-thirds. 
Thus,  classification  efforts  focus  on  technologies  that  can  reliably  differentiate  UXO  from  items 
that  can  be  safely  left  undisturbed. 

Classification  only  becomes  a  realistic  option  when  the  cost  of  identifying  items  that  may  be  left 
in  the  ground  is  less  than  the  cost  of  directly  digging  them.  Because  classification  generally 
requires  a  detection  survey  as  a  precursor  step,  the  investment  in  additional  data  collection  and 
analysis  must  result  in  sufficient  clutter  rejection  to  recuperate  the  investment.  Even  with  perfect 
detection  performance  and  high  signal-to-noise  ratio  (SNR)  values,  successfully  sorting  the 
detections  into  UXO  and  non-hazardous  items  is  a  difficult  problem  but,  because  of  its  potential 
payoff,  one  that  is  the  focus  of  significant  current  research.  This  demonstration  represents  an 
effort  to  transition  a  promising  classification  technology  into  widespread  use  at  UXO- 
contaminated  sites  across  the  country. 
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2.0  TECHNOLOGY 


2.1  TECHNOLOGY  DESCRIPTION 

Magnetic  and  EM  methods  represent  the  main  sensor  types  used  for  detection  of  UXO.  Over  the 
past  15  years,  significant  research  effort  has  been  focused  on  developing  methods  to  discriminate 
between  hazardous  UXO  and  non-hazardous  scrap  metal,  shrapnel  and  geology  (e.g.,  Bell  et  al., 
2001;  Pasion  et  al.,  2007;  Tantum  et  al.,  2008;  Liao  and  Carin,  2009).  The  most  promising 
classification  methods  typically  proceed  by  first  recovering  a  set  of  parameters  that  specify  a 
physics-based  model  of  the  object  being  interrogated.  For  example,  in  time-domain 
electromagnetic  (TEM)  data,  the  parameters  comprise  the  object  location  and  the  polarizability 
tensor  which  is  generally  decomposed  into  orientation  and  principal  polarizabilities.  Once  the 
parameters  are  recovered  by  inversion,  a  subset  of  the  parameters  is  used  as  feature  vectors  to 
guide  either  a  statistical  or  rule-based  classifier. 

There  are  three  key  elements  of  the  UXO  classification  process: 

1.  Creation  of  a  map  of  the  geophysical  sensor  data :  This  includes  all  actions  required  to 
form  an  estimate  of  the  geophysical  quantity  in  question  (i.e.,  amplitude  of  EMI  response 
at  a  given  time-channel)  at  each  of  the  visited  locations.  The  estimated  quantity  is 
dependent  on  the  following: 

a.  Hardware,  including  the  sensor  type,  deployment  platform,  position  and  orientation 
system  and  the  data  acquisition  system  used  to  record  and  time-stamp  the  different 
sensors; 

b.  Survey  parameters  such  as  line  spacing,  sampling  rate,  calibration  procedures  etc.; 

c.  Data  processing  such  as  merging  of  position/orientation  information  with  sensor 
data,  noise  and  background  filtering  applied; 

d.  The  background  environment  including  geology,  vegetation,  topography,  cultural 
features,  etc.;  and, 

e.  Depth  and  distribution  of  ordnance  and  clutter. 

2.  Anomaly  selection  and  feature  extraction :  This  includes  the  detection  of  anomalous 
regions  and  the  subsequent  extraction  of  a  polarization  tensor  model  for  each  anomaly. 
The  reliability  of  the  recovered  features  is  dependent  on  the  quality  of  the  survey  data.  If 
the  data  acquired  for  creation  of  the  data  map  is  not  of  sufficient  quality  to  extract  reliable 
parameters,  cued  data  acquired  in  a  static  mode  is  required.  Cued  data  has  higher  quality 
data  due  to  having  reduced  sensor  noise  by  being  acquired  in  a  static  mode,  and  by 
having  accurate  sensor  location  information  by  being  acquired  with  transmitters  and 
receivers  in  a  fixed  geometry. 

3.  Classification  of  anomalies:  The  final  objective  of  the  demonstration  is  the  production  of 
a  dig  sheet  with  a  ranked  list  of  anomalies.  This  will  be  achieved  via  classification 
algorithms  which  will  require  training  data  to  determine  the  attributes  of  the  UXO  and 
non-UXO  classes. 
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The  methodologies  for  data  processing,  feature  extraction,  and  statistical  classification  described 
above  have  been  implemented  within  the  UXOLab  software  environment,  which  was  used  for 
this  demonstration.  UXOLab  is  a  Matlab  based  software  package  developed  jointly  by  Black 
Tusk  Geophysics  (BTG)  and  University  of  British  Columbia  -  Geophysical  Inversion  Facility 
(UBC-GIF).  Key  modules  include  QCZilla  for  data  and  inversion  review,  TrainZilla  for 
identifying  training  data,  and  DigZilla  for  applying  statistical  and  library  based  classification 
techniques.  The  basic  data  processing  workflow  utilized  is  outlined  in  Figure  1. 


Figure  1.  Overview  of  Workflow  Applied  to  Each  Dataset  from  Spencer  Range 


Table  1  provides  a  list  of  sites  at  which  classification  methods  and  software  developed  at  the 
UBC-GIF  and  BTG  were  tested  and  demonstrated  as  part  of  the  ESTCP  program.  The  list 
summarizes  ESTCP  project  work  until  May  2014.  The  blue  text  indicates  projects  that  were 
ongoing  at  the  time  this  report  was  being  prepared.  Green  ‘X’s  indicate  data  sets  for  which 
industry  geophysicists  submitted  dig  lists  following  a  visit  to  BTG  for  training  on  classification 
theory  and  techniques.  Table  2  provides  additional  detail  for  a  selection  of  classification  studies 
completed  prior  to  the  Spencer  Range. 
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Table  1.  A  Summary  of  Sites  at  which  Classification  Has  Been  Implemented  by  BTG 
and  UBC-GIF  Personnel  as  Part  of  the  ESTCP  Munitions  Program 


1st 

Generation 

Vehicle 

Mounted 

Man 

Portable 

Dynamic 

SITE 

PROJECT 

MAG 

Geonics  EM61 

Geonics  EM63 

Q 

=> 

CD 

MetalMapper 

TEMTADS5X5 

BUD-HH 

AdlAI 

TEMTADS2X2 

MetalMapper 

AdlAI 

TEMTADS2X2 

Former  Lowry  Bombing  and  Gunnery  Range,  CO 

MM -0504 

X 

X 

X 

Former  Camp  Sibert,  AL 

MM -0504 

X 

X 

X 

X 

Former  Fort  McClellan, AL 

MM -0504 

X 

San  Luis  Obispo,  CA 

MM -0504 

X 

X 

X 

X 

X 

Former  Camp  Butner,  NC 

MR- 1004 

X 

X 

X 

Camp  Beale,  CA 

MR- 1004 

X 

X 

X 

X 

X 

Pole  Mountain,  WY 

MR-1159 

X 

X 

Spencer  Range,  TN 

MR-1158,59 

X 

X 

X 

X 

X 

X 

X 

Camp  Edwards,  MMR,  MA 

MR- 1226 

X 

X 

Camp  George  West,  CO 

MR- 1228 

X 

X 

Camp  Ellis,  IL 

MR- 1226 

X 

X 

Ft.  Rucker,  AL 

MR- 1226 

X 

New  Boston,  NH 

MR-1158,59 

X 

X 

X 

Southwest  Proving  Grounds,  AL 

MR-1226 

X 

X 

X 

X 

Waikaloa,  HI 

MR-1226 

X 

X 

Table  2.  Examples  of  Inversion  and  Classification  Testing  Completed  by  UBC-GIF 
and  BTG  as  Part  of  the  ESTCP  Program,  prior  to  the  Spencer  Range  Demonstration 


Site 

Camp  Sibert 

Sensors  and 

survey 

mode 

Geonics  EM-61  cart,  Multi-sensor  Towed  Array  Detection  System  (MTADS)  EM-61  array,  MTADS 
mag  array,  and  EM-63  single  and  cooperative  inversions.  EM-63  cued  interrogations  were 
positioned  by  a  Leica  TPS  1206  Robotic  Total  Station  (RTS)  with  orientation  information  provided 
by  a  Crossbow  AHRS400  Inertial  Measurement  Unit  (IMU). 

The  objective  of  the  surveys  was  the  classification  of  a  large  target  (4.2-in  mortars).  The  site  was 
unusual  in  that  the  primary  munition  known  to  have  been  used  was  the  4.2-in  mortar,  thus  providing 
a  site  where  the  classification  is  a  case  of  identifying  a  single  large  target  amongst  smaller  pieces  of 
mortar  debris  and  clutter. 

Description 

For  the  EM-61,  3-dipole  instantaneous  amplitude  models  were  fit  to  the  available  3  time-channels, 
while  for  the  EM-63,  3-dipole  Pasion-Oldenburg  models  were  recovered  from  the  26  time-channel 
data.  MTADS  and  EM-63  data  were  also  cooperatively  inverted.  Parameters  of  the  dipole  model 
were  used  to  guide  a  statistical  classification. 

Results 

The  results  for  all  sensor  combinations  were  excellent,  with  just  one  false  negative  for  the  EM-63 
when  inverted  without  cooperative  constraints.  When  inverted  cooperatively,  the  EM-63  cued 
interrogation  was  the  most  effective  discriminator.  All  33  UXO  were  recovered  with  25  false  alarms 
(16  of  these  were  in  the  "can't  analyze"  category).  Not  counting  the  "can't  analyze"  category,  the  first 
33  recommended  excavations  were  all  UXO. 

The  MTADS  and  MTADS  cooperatively  inverted  were  also  very  effective  at  discrimination,  with  all 
UXO  recovered  very  early  in  the  dig  list.  The  MTADS  data  set  suffered  from  a  high  number  of  false 
alarms  due  to  anomalies  with  a  geological  origin.  In  addition,  the  operating  point  was  very 
conservative  and  many  non-UXO  were  excavated  after  recovery  of  the  last  UXO  in  the  dig  list. 

The  results  from  the  EM-61  cart  were  also  very  good,  although  24  false-positives  were  required  to 
excavate  all  105  UXO.  The  lower  data  quality  of  the  EM-61  cart  resulted  in  a  larger  number  of  "can't 
analyze"  anomalies  over  metallic  sources  than  the  MTADS. 
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Table  2.  Examples  of  Inversion  and  Classification  Testing  Completed  by  UBC-GIF  and 
BTG  as  Part  of  the  ESTCP  Program,  prior  to  the  Spencer  Range  Demonstration 

(Continued) 


Site 

San  Luis  Obispo 

Sensors  and 

survey 

mode 

Detection  mode  surveys  included:  MTADS  magnetometer  and  EM-61  arrays;  Geonics  EM-61  cart; 
and  Man-portable  Simultaneous  EMI  and  Magnetometer  System  (MSEMS)  cart.  Cued  interrogation 
mode  TEMTADS,  MetalMapper  and  BUD  surveys  were  also  conducted. 

At  this  site  the  objective  was  to  identify  TOI  from  a  number  of  different  target  classes:  primarily  60- 
mm,  81-mm,  4.2-in  mortars,  2.36-in  rockets,  and  one  each  of  37-mm,  3-in  and  5-in  projectiles.  The 
site  had  significant  topographic  relief. 

Description 

For  all  TEM  data  sets,  3 -dipole  instantaneous  amplitude  models  were  fit  to  the  available  time- 
channels.  Single  dipoles  were  fit  to  targets  in  magnetics  data  sets,  but  cooperative  inversion  was  not 
used  at  this  site.  For  detection  data  sets,  a  threshold  on  the  rate  of  decay  of  primary  polarizabilities 
was  used  to  rank  targets. 

Dig  sheets  for  cued  interrogation  data  sets  were  generated  using  statistical  classifiers  trained  on 
size/decay  features,  as  well  as  with  library  methods  and  an  "expert"  method  based  on  the  judgment 
of  an  analyst. 

Results 

Magnetometer  detection  and  classification  performance  at  this  site  was  quite  poor.  The  EM-61 
production  datasets  were  much  more  effective  than  magnetics.  Size  estimated  from  the  recovered 
polarizations  was  not  an  effective  classification  metric  due  to  the  small  size  of  the  60-mm  mortars 
and  the  inability  to  accurately  constrain  depth.  However,  the  time-decay  rate  estimated  from  the 
recovered  polarizabilities  provided  an  effective  ranking  scheme.  The  EM-61  cart  performance  was 
marginally  better  than  the  MSEMS  cart  and  MTADS  EM-61  array. 

For  the  TEMTADS  data  the  library  method  was  the  most  effective  with  204  of  206  TOI  recovered 
along  with  131  of  1076  non-TOI.  The  two  false  negatives  were  the  rocket  motor  pieces  declared 
non-TOI  by  all  cued-interrogation  methods  and  a  60-mm  mortar  with  a  target  response  that 
overlapped  with  some  nearby  clutter.  The  other  classification  methods  were  also  effective, 
generating  between  2  to  4  false-negatives. 

The  library  method  was  again  most  effective  for  the  MetalMapper  data  with  the  excavation  of  203  of 
204  TOI  and  175  of  1205  non-TOI.  Correct  classification  of  ordnance  type  was  also  achieved  with 
up  to  99%  accuracy  achieved  with  statistical  classification. 

Site 

Camp  Butner 

Sensors  and 

survey 

mode 

Detection  surveys  with  the  EM-61  cart  and  MetalMapper  dynamic  sensors  were  conducted.  Detected 
targets  from  the  EM-61  data  were  revisited  with  MetalMapper  static  and  TEMTADS  cued 
interrogations. 

TOI  at  the  site  included  105mm  high-explosive  anti-tank  (HEAT),  37mm,  and  M48  fuses.  Topographic 
relief  was  benign,  but  there  was  a  significant  amount  of  clutter  similar  in  size  and  shape  to  37mm. 

Description 

For  all  TEM  data  sets,  3 -dipole  instantaneous  amplitude  models  were  fit  to  the  available  time- 
channels.  For  detection  data  sets,  a  threshold  on  the  rate  of  decay  of  primary  polarizabilities  was 
used  to  rank  targets. 

Dig  sheets  for  cued  interrogation  data  sets  were  generated  using  statistical  classifiers  trained  on  size/ 
decay  features,  as  well  as  with  library  methods  and  an  "expert"  method  based  on  the  judgment  of  an 
analyst. 

Results 

Thresholding  on  time-decay  rate  of  estimated  polarizabilities  estimated  from  EM-61  data  performed 
quite  poorly,  with  a  0.92  false  alarm  rate  (FAR).  This  is  likely  attributable  to  poor  depth  estimation 
for  small  targets.  The  MetalMapper  dynamic  data  produced  reliable  depth  estimates,  but  did  not 
measure  sufficiently  late  in  time  to  provide  separation  between  TOI  and  non-TOI  polarizabilities. 

FAR  was  approximately  0.7  for  this  sensor. 

Excellent  classification  performance  was  achieved  with  the  TEMTADS  data:  all  171  TOI  were 
found  with  a  FAR  of  only  5%.  MetalMapper  static  performance  was  similar  to  TEMTADS,  but 
produced  a  much  higher  FAR  (78%)  owing  to  outlying  TOI  attributable  to  faulty  data. 
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Table  2.  Examples  of  Inversion  and  Classif  ication  Testing  Completed  by  UBC-GIF  and 
BTG  as  Part  of  the  ESTCP  Program,  prior  to  the  Spencer  Range  Demonstration 

(Continued) 


Site 

Pole  Mountain  WY. 

Sensors  and 

survey 

mode 

Detection  mode  survey  was  EM-61. 

Cued  interrogation  mode  MetalMapper  survey  was  also  conducted. 

Description 

Dipole  models  were  fit  to  the  available  time-channels.  Multi-object  inversions  were  also  carried  out 
on  the  MetalMapper  data  set. 

Results 

Excellent  quality  MetalMapper  data  resulted  in  near  perfect  classification  results  with  all  TOI 
recovered  with  less  than  5%  non-TOI  dug. 

Detection  mode  EM-61  data  had  limited  detection  ability. 

Site 

Camp  Beale 

Sensors  and 

survey 

mode 

Detection  mode  survey  was  EM-61. 

Cued  interrogation  mode  MetalMapper  surveys  were  conducted.  Also,  the  first  test  of  portable 
sensors  including  MPV,  TEMTADS  2x2,  Handheld  BUD. 

The  site  had  magnetic  soil  present  which  reduced  the  SNR  of  the  data  relative  to  previous  sites. 

Description 

For  all  TEM  data  sets,  dipole  polarizabilities  were  fit  to  the  available  time-channels.  Multi-object 
inversions  were  also  carried  out  on  all  data  sets. 

For  EM-61  detection  data,  the  polarizability  rate  of  decay  and  size  were  used  to  rank  targets. 
Anomalies  collected  by  MetalMapper  and  portable  sensors  were  ranked  using  statistical  and  library- 
based  classification  on  polarizabilities. 

Results 

Similar  classification  performance  was  achieved  for  all  portable  sensors.  For  each  portable  sensor 
excellent  classification  performance  was  achieved  with  no  false  negatives. 

MetalMapper  diglists  also  had  excellent  performance.  A  diglist  featuring  no  false  negatives  was 
submitted  for  both  sets  of  MetalMapper  data. 

2.2  ADVANTAGES  AND  LIMITATIONS  OF  THE  TECHNOLOGY 

The  main  advantage  of  the  technology  is  a  potential  reduction  in  the  number  of  non-hazardous 
items  that  need  to  be  excavated,  thus  reducing  the  costs  of  UXO  remediation.  Advantages  of 
UXOLab  and  the  algorithms  within  the  package  include: 

•  All  the  functionality  required  to  process  raw  geophysical  data,  detect  anomalous  regions, 
and  perform  geophysical  inversion  and  classification. 

•  Algorithms  for  inverting  magnetic  and  TEM  data  sets  both  separately  and  cooperatively 
using  a  number  of  different  polarization  tensor  formulations. 

•  Extensive  set  of  algorithms  for  rule-based  and  statistical  classification  algorithms. 

•  Configuration  in  a  modular  fashion,  so  that  as  new  sensor  technologies  become  available 
(e.g.  new  TEM  systems  with  multi-component  receivers  etc),  the  inversion  functionality 
will  be  immediately  available  to  those  new  sensor  systems. 

•  Intuitive  design  and  user-friendly  GUIs  and  workflows 
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The  principal  disadvantage  is  that  UXOLab  has  not  been  configured  for  general  use  by  contractors 
and  non-specialists.  However,  as  part  of  ESTCP  MR-201004,  we  transitioned  our  inversion 
algorithms  to  an  Application  Program  Interface  (API)  that  enables  access  of  the  algorithms  to  other 
software. 
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3.0  PERFORMANCE  OBJECTIVES 


The  performance  objectives  for  this  demonstration  are  summarized  in  Table  3.  The  first  three 
analysis  objectives  refer  to  the  classification  part  of  the  demonstration  with  the  first  two  referring 
to  the  best  results  from  each  approach  in  a  retrospective  analysis  and  the  third  addressing  how 
well  each  demonstrator  is  able  to  specify  the  correct  threshold  in  advance.  The  final  two 
objectives  refer  to  the  accuracy  of  target  features  extracted  from  the  data. 

Table  3.  Description  of  Performance  Objectives  for  the  Spencer  Range  Demonstration 


Performance 

Objective 

Metric 

Data  Required 

Success  Criteria 

Maximize  correct 
classification  of 
TOI 

Number  of  TOI  retained 

•  Prioritized  anomaly  lists 

•  Scoring  reports  from 

Institute  for  Defense 

Analyses  (IDA) 

Approach  correctly 
classifies  all  TOI 

Maximize  correct 
classification  of 
non-TOI 

Number  of  false  alarms 
eliminated 

•  Ranked  anomaly  lists 

•  Scoring  reports  from  IDA 

Reduction  of  clutter  digs 
by 

>75%  while  retaining  all 
TOI 

Specification  of 
no-dig  threshold 

Probability  of  correct 
classification  of  TOI  and 
number  of  false  alarms  at 
demonstrator  operating 
point 

•  Demonstrator- specified 
threshold 

•  Scoring  reports  from  IDA 

Threshold  specified  by  the 
demonstrator  to  achieve 
criteria  above 

Minimize  number 
of  anomalies  that 
cannot  be 
analyzed 

Number  of  anomalies  that 
must  be  classified  as 
“Unable  to  Analyze” 

•  Demonstrator  target 
parameters 

Reliable  target  parameters 
can  be  estimated  for  >95% 
of  anomalies  on  each 
sensor’ s  detection  list. 

Correct 
estimation  of 
target  parameters 

Accuracy  of  estimated 
target  parameters  for  seed 
items 

•  Demonstrator  target 
parameters 

•  Results  of  intrusive 
investigation 

Polarizabilities  ±20% 

X,  Y  <15  cm  (la) 

Z  <10  cm  (la) 

Table  4  summarizes  the  performance  of  the  different  sensor  modes  in  the  Open  and  Treed  areas 
while  Table  5  covers  summarizes  the  dynamic  area.  All  performance  metrics  except  two  were 
met:  for  the  MetalMapper  (URS)  dynamically  acquired  data,  reliable  target  parameters  were 
estimated  for  only  92.1%,  due  to  the  lack  of  data  coverage  at  the  edges  of  the  survey  area.  The 
data  acquired  in  the  Treed  area  by  the  TEMTADS  2x2  had  a  depth  estimate  error  standard 
deviation  of  13  cm  due  to  variations  in  the  ground  clearance  height  in  that  area. 
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Table  4. 


Summary  of  Performance  Objectives  for  the  Open  and  Treed  Areas  of  the 
Spencer  Range  Demonstration. 


Performance 

Objective 

Metric 

Success  Criteria 

Data  Set 

Open  Area 

Treed  Area 

Metal-Mapper 
URS  -  Cued 

Metal-Mapper 
Naeva  -  Cued 

TEMTADS  2x2  - 
Cued 

Maximize 

correct 

classification 
of  TOI 

Number  of 
TOI  retained 

Approach 
correctly  classifies 
all  TOI 

100% 

100% 

100% 

Maximize 

correct 

classification 

ofnon-TOI 

Number  of 
lalse  alarms 
eliminated 

Reduction  of 
clutter  digs  by  > 
75%  while 
retaining  all  TOI 

88.5% 

84.8% 

84.7% 

Specification 
of  no-dig 
threshold 

Probability 
of  correct 
classificatio 
n  of  TOI 

and  number 
of  lalse 

alarms  at 
demonstrate 
r  operating 
point 

Threshold 
specified  by  the 
demonstrator  to 

achieve  criteria 
above 

P  =1 

*  class  1 

Nfa  =185 
(18.2%) 

P  =1 

*  class  1 

N&  =  209 
(20.5%) 

P  =1 

N&=  106(17.1%) 

Minimize 
number  of 
anomalies  that 
cannot  be 
analyzed 

Number  of 

anomalies 
that  must  be 
classified  as 
“Unable  to 
Analyze” 

Reliable  target 
parameters  can  be 
estimated  for  > 

95%  of  anomalies 
on  each  sensor’s 
detection  list. 

98.8% 

99.7% 

99.2% 

Correct 

estimation  of 

target 

parameters 

Accuracy  of 
estimated 
target 
parameters 
for  seed 

items 

Polarizabilities  - 

/+20% 

X,  Y<  15cm(lo) 
Z  <  10  cm  (let) 

N(>20%)=4 

N(<20%)=82 

<yx=0.08m 

ay=0_09m 

az=0.05m 

N(>20%)=6 

N(<20%)=84 

<rx=0.09m 

ay=0.098m 

az=0.05m 

N(>20%)=4 

N(<20%)=70 

ax=NA 

ay=NA 

ar=0.13m 
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Table  5.  Summary  of  Performance  Objectives  for  the  Dynamic  Area  of  the  Spencer 

Range  Demonstration. 


Data  Set 

Dynamic  Area 

Performance 

Objective 

Metric 

Success  Criteria 

Metal- 

Mapper 

URS- 

Dynamic 

TEMTADS 
2x2  - 
Cued 

TEMTADS 
2x2  - 
Dynamic 

Metal- 

Mapper 

URS- 

Cued 

Maximize 

correct 

classification 

ofTOI 

Number  of 
TOI  retained 

Approach 
correctly  classifies 

all  TOI 

100% 

100% 

100% 

100% 

Maximize 

correct 

classification 
of  non-TOI 

Number  of 
false  alarms 
eliminated 

Reduction  of 
clutter  digs  by  > 
75%  while 
retaining  all  TOI 

83.5% 

91.5% 

82.0% 

95.3% 

Specification 
of  no-dig 
threshold 

Probability 
of  correct 
classificatio 
n  of  TOI 
and  number 
of  false 
alarms  at 
demonstrato 
r  operating 
point 

Threshold 
specified  by  the 
demonstrator  to 
achieve  criteria 
above 

Pclass  —  1 

Nfa  =  67 
(21.2%) 

Eclass  —  1 

Nfa  -  44 
(13.9%) 

Eclass  —  1 

Nfa  =  75 
(23.7%) 

Eclass  —  1 

Nfa  =  35 
(11.1%) 

Minimize 
number  of 
anomalies  that 
cannot  be 
analyzed 

Number  of 
anomalies 
that  must  be 
classified  as 
“Unable  to 
Analyze” 

Reliable  target 
parameters  can  be 
estimated  for  > 

95%  of  anomalies 
on  each  sensor’s 
detection  list. 

92.1% 

100% 

100% 

100% 

Correct 
estimation  of 
target 
parameters 

Accuracy  of 

estimated 

target 

parameters 

for  seed 

items 

Polarizabilities  - 
/+20% 

X,  Y  <  15  cm  (Ict) 

Z  <  10  cm  (lcr) 

N(>20%)=3 

N(<20%)=18 

ax=0.09m 

cyy=0.06m 

az=0.03m 

N(>20%)=4 

N(<20%)=22 

ax=NA 

ay=NA 

crz=0.08m 

N(>20%)=7 

N(<20%)=16 

ax=0.07m 

ay=0.07m 

crz=0.07m 

N(>20%)=3 

N(<20%)=20 

ax=0.07m 

ay=0.07m 

az=0.05m 
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4.0  SITE  DESCRIPTION 


4.1  SITE  LOCATION  AND  HISTORY 

Located  in  Spencer/Van  Buren  County,  Tennessee,  the  30,618  acre  Spencer  Artillery  Range  was 
established  in  1941,  and  documentation  identifies  two  impact  areas:  Jakes  Mountain  (5,060 
acres)  and  Bald  Knob  (2,090  acres).  Troop  training  took  place  until  September  1944.  Subsequent 
arrangements  were  made  for  Dyersburg  Army  Air  Field  to  use  the  Spencer  Artillery  Range  as  an 
air-to-ground  gunnery  range.  The  site  contains  partially  wooded  and  cleared  areas.  This 
demonstration  will  be  conducted  on  three  discrete  areas  within  the  former  range:  a  3.7-acre 
wooded  area,  a  4.4-acre  open  field  area,  and  a  1.3-acre  dynamic  area. 

4.2  SITE  GEOLOGY 

Details  of  site-geology  can  be  found  in  the  ESTCP  (2014)  summary  report  for  this  demonstration 
site. 


4.3  MUNITIONS  CONTAMINATION 

Munitions  expected  at  the  site  include  37mm,  75mm,  76mm,  105mm,  and  155mm  projectiles. 
Additional  details  regarding  munitions  contamination  can  be  found  in  the  ESTCP  (2014) 
summary  report  for  this  demonstration  site 
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5.0  DATA  ANALYSIS  AND  PRODUCTS 


See  the  ESTCP  (2014)  summary  report  for  a  description  of  the  test  design  for  the  overall  project. 

BTG/UBC-GIF  processed  data  and  delivered  the  following  two  prioritized  dig  lists  for  the 
Spencer  Range  Open  Area: 

1)  Cued  URS  MetalMapper  library  match:  A  dig  sheet  was  produced  based  on  how  well  the 
recovered  polarizabilities  matched  the  polarizabilities  in  a  library  of  ordnance  items 
expected  at  the  site;  and 

2)  Cued  NAEYA  MetalMapper  automated  DigZilla  ranking:  A  dig  sheet  was  produced 
based  on  a  weighted  sum  of  fit  to  the  library,  polarizability  size,  polarizability  decay,  and 
a  measure  of  how  "rod-like"  the  target  is  based  on  secondary  polarizabilities. 

BTG/UBC-GIF  processed  data  and  delivered  the  following  prioritized  dig  list  for  the  Spencer 
Range  Tree  Area: 

1)  Cued  TEMTADS  2x2  library  match:  A  dig  sheet  was  produced  based  on  how  well  the 
recovered  polarizabilities  matched  the  polarizabilities  in  a  library  of  ordnance  items 
expected  at  the  site. 

BTG/UBC-GIF  processed  data  and  delivered  the  following  four  prioritized  dig  lists  for  the 
Spencer  Range  Dynamic  Area: 

1)  Cued  URS  MetalMapper  library  match:  Same  method  as  Open  Area; 

2)  Dynamic  URS  MetalMapper  Combined  Classifier  Ranking  (CCR)  match:  A  dig  sheet  is 
submitted  by  combining  separate  ranked  lists  based  on  polarizability  misfit,  size,  and 
decay; 

3)  Cued  TEMTADS  2x2  library  match:  Same  method  as  Tree  Area;  and 

4)  Dynamic  TEMTADS  2x2  CCR:  Same  as  in  3)  but  for  the  dynamic  TEMTADS  2x2  data. 
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6.0  DATA  ANALYSIS  AND  PRODUCTS 


The  Spencer  Range  demonstration  site  was  divided  up  into  three  distinct  areas  differentiated  by 
the  sensors  deployed:  “Open”  area,  “Treed”  area,  and  “Dynamic”  area.  The  Open  area  was 
surveyed  in  a  cued,  static  mode  by  the  MetalMapper.  Cued  MetalMapper  data  were  acquired  by 
NAEVA.  and  URS  using  the  same  anomaly  list.  A  more  challenging,  treed  survey  area  was 
surveyed  by  cued  portable  sensors  including  the  HandHeld  BUD,  MPV,  and  TEMTADS  2x2. 
There  was  also  a  relatively  small  dynamic  area  that  was  surveyed  by  the  TEMTADS  2x2,  MPV 
and  MetalMapper  in  both  cued  and  dynamic  modes  to  provide  cued  vs.  dynamic  deployment 
comparison  for  identical  targets. 

BTG  processed  multiple  datasets  in  order  to  test  practical  UXO  classification  methods  over  a 
range  of  EMI  instruments  deployed.  BTG  processed  all  the  Open  area  datasets,  the  TEMTADS 
2x2  and  MPV  data-sets  from  the  Treed  area,  and  all  the  TEMTADS  and  MetalMapper  URS  data 
from  the  Dynamic  area.  All  MPV  processing  was  carried  out  under  ESTCP  MR- 201 158,  and  all 
MPV  results  are  reported  under  that  project. 

The  basic  data  processing  workflow  was  the  same  for  all  datasets  and  is  outlined  in  Figure  1. 
Classification  was  performed  by  BTG  or  University  of  British  Columbia  (UBC)  analysts  with  a 
strict  firewall  maintained  throughout  the  analyses.  Due  to  this  firewall,  information  that  could  aid 
classification  for  a  particular  dataset  could  NOT  be  transferred  from  another  dataset  (for 
example,  classes  of  potential  TOI). 

In  the  interest  of  brevity,  in  this  report  we  will  only  describe  the  data  analysis  procedures  used 
for  one  of  the  datasets:  MetalMapper  cued  data  (URS)  from  the  Open  Area.  The  work-flows  for 
the  other  datasets  were  very  similar  with  additional  details  available  in  Pasion  et  al.  (2013). 

6.1  PREPROCESSING 

MetalMapper  cued  data  for  all  anomalies  were  received  as  a  set  of  raw  TEM  files  and  two  sets  of 
CSV  files  (with  and  without  background  corrections).  Our  analyses  used  the  background- 
corrected  data. 

6.2  TARGET  SELECTION  FOR  DETECTION 

Target  detection  lists  were  provided  by  the  Program  Office. 

6.3  PARAMETER  ESTIMATES 

The  data  were  inverted  in  UXOLab  using  a  sequential  inversion  approach  to  estimate 
target  location,  depth  and  principal  polarizabilities.  Instrument  height  above  the  ground  was 
assumed  to  be  10  cm.  Noise  standard  deviation  estimates  were  not  available,  so  a  constant 
noise  value  of  1  over  all  time  channels  was  used.  Target  location  was  constrained  to  lie 
between  ±0.5  meter  (m)  in  both  X  and  Y  directions  relative  to  the  picked  location.  Target 
depth  was  constrained  to  lie  between  -1.2  and  0  m.  The  initial  optimization  for  target  location 
identified  up  to  three  starting  models  to  input  into  the  subsequent  estimation  of  polarizabilities. 
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We  performed  two  inversions  per  anomaly,  solving  for:  (1)  a  single  object  (single  object 
inversion  [SOI]);  and  (2)  two  objects  (two  object  inversion  [201]). 

Analysis  of  the  data,  including  visual  quality  control  (QC)  of  data  and  model  parameters, 
selection  of  training  data,  and  dig  list  creation,  was  performed  using  the  UXOLab  software  suite. 
Visual  QC  of  the  data  was  performed  using  QCZilla ,  which  provides  a  thorough  overview  of  the 
observed  and  predicted  data,  predicted  model  parameters,  and  measures  of  data/model  quality. 
Display  of  the  gridded  EM-61  data  at  each  anomaly  provides  a  useful  indicator  of  the  anomaly 
size  and  strength.  Predicted  polarizabilities  were  compared  to  reference  polarizabilities  for 
various  ordnance  items  initially  derived  from  instrument  verification  strip  (IVS)  measurements. 
The  Spencer  Artillery  Range  test  pit  contained  four  items:  75-mm  projectiles,  37-mm  projectiles. 
Small  ISO,  and  Shot  Put.  The  latter  item  was  not  used  during  the  classification  process.  As  the 
analysis  proceeded,  the  library  of  reference  items  was  augmented  with  additional  items  based  on 
ground  truth  obtained  through  training  data  requests.  Each  item  in  the  ordnance  reference  library 
was  assigned  a  size  (diameter)  in  mm.  Each  item  classified  as  "likely  TOI”  in  the  submitted  dig 
list  was  assigned  a  size  based  on  the  ordnance  item  in  the  reference  library  with  the  best 
matching  primary  polarizability  (Li). 

During  data/model  QC  the  primary  objectives  were  to  (1)  flag  high-likelihood  TOI;  and  (2)  fail 
bad  models  and  inversions.  Anomalies  flagged  as  high-likelihood  TOI  were  monitored  during 
the  dig  list  creation  phase  to  ensure  they  were  being  dug,  ideally  early  in  the  dig  list.  Models  and 
inversions  were  considered  to  be  bad  when  the  inversion  failed  (i.e.,  the  data  misfits  are  large),  or 
when  the  recovered  model  location(s)  were  on,  or  near,  an  inversion  boundary.  With  multi-object 
inversions,  it  is  not  uncommon  that  one  of  the  models  is  unrealistic  (e.g.,  deep,  large  in 
magnitude,  sometimes  located  on  or  near  a  horizontal  inversion  boundary)  yet  provides  the  best 
fit  to  the  reference  polarizabilities  (e.g.,  Figure  2).  In  all  of  these  cases,  the  model  was  flagged  as 
failed.  Models  flagged  as  failed  were  not  used  in  the  classification  process.  Anomalies  with  all 
models  from  all  inversions  failed  were  classified  as  "can't  extract  reliable  parameters";  these 
anomalies  will  be  dug.  For  a  given  anomaly,  if  more  than  one  model  was  passed  the 
classification  procedure  will  consider  all  passed  models  and  effectively  use  the  one  that  is  "best". 

The  Spencer  Open  Area  MetalMapper  Cued  dataset  comprised  1104  unique  anomalies.  Of  the 
3312  total  models,  2623  were  passed  and  used  in  the  classification  process;  689  were  failed. 
Thirteen  anomalies  were  classified  as  “cannot  extract  reliable  parameters”  due  to  poor 
inversion  results.  One  of  these  (anomaly  308)  corresponded  to  a  TOI  (37mm).  102  anomalies 
were  classified  as  “high  likelihood  UXO"  during  QC;  83  of  these  (81%)  correspond  to  actual 
TOI.  The  total  number  of  unique  TOI  in  the  Spencer  Open  Area  is  86. 
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Target:  1250  (Cell  1088) 

Tag:  SR_F_01 250_spn_T00920_01 250.CSV 
Spencer_MMcued_URS_S0l_20l 

lnv#1/2=SOI:  1/1  (pass) 


T5X 


-0.5 


-1 


-1  0  1  Chi 


RX-TX  RY-TX  RZ-TX  RX-TY  RY-TY  RZ-TY  RX-TZ  RY-TZ  RZ-TZ 


Decay  vs.  Size 


201 


Figure  2.  Example  of  an  Unrealistic  201  Model  (anomaly  1250;  frag) 


The  first  model  of  the  201  (model  2)  provides  the  best  fit  (i.e.,  minimum  misfit)  to  the  reference 
polarizabilities  (misfit  =  0.826),  but  the  predicted  depth  of  1.2  m,  the  location  at  the  edge  of  the 
instrument  (i.e.,  at  the  vertical  and  horizontal  inversion  boundaries),  and  the  high  amplitude  and  jittery 
appearance  of  the  polarizabilities,  are  classic  signs  that  this  model  is  an  artifact  of  the  multi-object 
inversion  process.  Accordingly,  this  model  was  failed  during  QC.  Polarizabilities  for  SOI  and  201  are 
shown  at  left.  Modeled  target  locations  (X-Y  and  Z)  are  shown  at  the  top  right  (gridded  EM-61  data  is 
displayed  behind  the  X-Y  plot).  Gridded  observed,  predicted  and  residual  data  for  SOI  and  201  are 
shown  below  location  maps.  Decay  versus  size  feature  plot  is  shown  in  bottom  right.  Dots  are  test  data; 
stars  are  reference  items.  Numbered  circles  are  models  for  this  anomaly. 
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6.4  CLASSIFIER  AND  TRAINING 


Figure  3  shows  the  distribution  of  passed  models  in  decay  versus  size  feature  space. 
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Figure  3.  Distribution  of  Passed  Models  in  Decay(tl,t29)  versus  Size(tl)  Feature  Space 

where  size(tl )  is  the  total  polarizability  measured  at  the  first  time  channel  ( tl=0.106  milliseconds),  and 
decay(tl,t29)  is  size(tl)/size(t29)  where  t29=2.006  ms.  Some  outliers  are  not  shown.  Labeled  stars 

represent  ordnance  library  reference  items. 

Our  analysis  method  is  based  primarily  on  polarizability  matching  with  respect  to  ordnance  items 
in  a  reference  library.  For  this  approach  to  be  successful,  it  is  important  to  determine  the  types  of 
ordnance  present  at  the  site.  During  visual  QC  the  analyst  keeps  track  of  suspicious,  UXO-like 
items  (i.e.,  items  with  modeled  polarizabilities  possessing  UXO-like  properties).  Training  data 
for  some  of  these,  particularly  those  with  polarizabilities  different  from  the  items  in  the  reference 
library,  would  be  requested.  In  addition,  we  used  our  custom  training  data  selection  tool, 
TrainZilla,  to  explore  feature  space  and  automatically  search  for  clusters  of  items  with  self¬ 
similar  polarizabilities.  In  TrainZilla ,  the  user  selects  a  region  in  feature  space  by  drawing  a 
polygon,  and  the  program  automatically  identify  clusters  of  self-similar  feature  vectors  by 
computing  a  misfit  matrix  M  with  elements 

MjK  =  ZiLi ip 
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where  LJtotal  is  the  log-transformed  total  polarizability  for  the  jth  feature  vector.  Feature  vectors 
with  mutual  misfit  less  than  a  user-specified  threshold  define  a  cluster  in  polarizability  space. 
This  analysis  helps  to  identify  clusters  that  may  not  be  readily  evident  in  decay-size  feature 
space:  e.g.,  targets  with  consistent  polarizabilities  that  may  be  hidden  in  the  “cloud”  of  non- 
TOI  features.  A  basic  example  of  the  use  of  TrainZilla  is  shown  in  Figure  4  and  Figure  5. 


Polygon  #  1  -  cluster  #  1  of  1 :  7  feature  vectors 


Size 

Figure  4.  Example  of  Use  of  the  Training  Data  Selection  Tool  ( TrainZilla ) 

A  polygon  ( solid  black  line)  is  drawn  in  feature  space.  Clusters  of  items  with  self-similar  polarizabilities 
are  automatically  found  based  on  the  specified  cluster  search  parameters.  In  this  case  a  cluster 
comprising  7  features  is  visible  (solid  feature  symbols  encompassed  by  broken  line).  Polarizabilities  for 

this  cluster  are  shown  in  Figure  5. 

Our  training  data  requests  typically  focused  on:  (1)  items  whose  polarizabilities  exhibited  UXO- 
like  properties  distinct  from  those  of  items  in  our  reference  library;  (2)  items  with  polarizabilities 
similar  to  items  in  our  reference  library,  but  with  degraded  quality;  and  (3)  one-off  items.  Figure 
6  shows  the  location  in  decay-size  feature  space  of  all  training  requests.  We  paid  particular 
attention  to  items  with  polarizabilities  suggestive  of  small  objects  such  as  fuses  and  small  caliber 
projectiles.  All  of  these  turned  out  to  be  non-TOI. 

In  our  initial  training  request  for  49  items,  twelve  of  these  were  TOI,  two  of  which  were 
previously  unknown  ordnance  items:  60mm  mortar  and  medium  ISO.  We  subsequently 
requested  training  on  two  more  items,  both  of  which  proved  to  be  non-TOI. 
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Polygon  1  -  Cluster  1  (7  pols) 
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Figure  5.  Polarizabilities  for  the  Cluster  Shown  in  Figure  4 

Colored  lines  are  predicted  polarizabilities.  Broken  grey  lines  are  best  fitting  reference  polarizabilities. 
Training  data  were  requested  for  Anomalies  897  and  1097  (stars);  ground  truth  (photos)  revealed  that 
both  of  these  anomalies  are  new  TOI  ( relative  to  our  starting  ordnance  reference  library  which 
comprised  the  four  items  from  the  test  pit).  Anomaly  897  is  60mm  mortar;  Anomaly  1097  is  a  medium 
ISO.  These  items  were  added  to  the  reference  library. 

6.4.1  Classification  Method 

Our  dig  lists  were  developed  using  the  DigZilla  module  of  UXOLab  (Figure  7).  DigZilla  allows 
for  the  creation  of  multi-stage  dig  lists  with  minimal  effort,  and  supports  a  number  of  classifiers. 

Our  initial  dig  list  comprised  two  stages.  For  the  early  digs  (1-201)  the  order  was  based  on 
polarizability  misfit  using  all  three  polarizabilities.  The  second  stage  (digs  202-1104)  was  based 
on  polarizability  misfit  using  only  the  primary  polarizability.  Figure  8  contains  the 
polarizabilities  used  for  classification. 
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Figure  6.  Decay  versus  Size  Feature  Space  Plot  Showing  Passed  Models  (blue  dots)  and 
the  Location  of  Training  Data  Requests  (red  dots) 

Stars  are  library  reference  items. 


Figure  7.  Screen  Shot  of  the  DigZilla  Graphical  User  Interface 

Features  in  the  decay  versus  size  feature  plot  are  color  coded  according  to  dig  list  order. 
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1 : 37mm 


2: 75mm 


3:  Small  ISO  2 


4: 105mm 


13: 37mm-d 


14:  Unknown 


15:  Unknown-2 


Figure  8.  Ordnance  library  for  the  MetalMapper  URS  Cued  Data 

Polarizabilities  1  to  13  were  used  for  the  Open  area.  Polarizabilities  14  and  15  were  added  processing 

cued  data  acquired  in  the  Dynamic  area. 
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7.0  PERFORMANCE  ASSESSMENT 


Table  6  and  Table  7  summarize  the  classification  performance  for  all  of  the  data  sets.  The 
classification  metrics  were  calculated  using  the  ground  truth  file  delivered  by  the  ESTCP 
program  office  and  the  final  submitted  dig  list  for  each  dataset.  The  model  fit  statistics  (i.e., 
polarizability  percent  error,  estimated  location  error,  and  depth  error)  were  calculated  using  all 
anomalies  collected.  Therefore,  if  multiple  data  soundings  were  acquired  over  the  same  target, 
the  resulting  best-fit  model  from  each  sounding  was  included  in  the  calculations.  For  the 
TEMTADS  2x2,  the  location  errors  were  not  recorded  due  to  the  absence  of  IMU  information. 
The  IMU  information  is  required  for  estimating  the  dipole  source  location  (and  therefore  target 
location)  in  geographic  coordinates. 


Table  6.  Classification  Results  from  Open  and  Tree  areas  of  Spencer  Range 


Open  Area 

Trees 

Instrument 

Metal  Mapper  URS 

Metal  Mapper 

NAEVA 

TEMTADS  2x2 

Site 

Open 

Open 

Trees 

Survey  Type 

Cued 

Cued 

Cued 

Analyst 

Zelt 

Kingdon 

Song 

Anomaly 

Statistics 

Total  Anomalies 

1104 

1104 

690 

Num  TOI 

86 

86 

71 

Num  Non-TOI 

1018 

1018 

619 

FAR 

Percent  of  Non-TOI 

0.115 

0.152 

0.153 

Num  Non-TOI  Dug 

117 

155 

95 

Digs 

271  (24.5%) 

295  (26.7%) 

177  (25.7%) 

Stop  Dig  Point 

TOI  Digs 

86  (100.0  %) 

86  (100.0  %) 

71  (100.0  %) 

Non-TOI  Digs 

185  (18.2  %) 

209  (20.5  %) 

106(17.1  %) 

Digs 

51 

43 

63 

Training  Data 

TOI 

12 

8 

4 

Non-TOI 

39 

35 

59 

"Cant 

Digs 

13 

3 

5 

Analyze" 

TOI 

1 

0 

0 

anomalies 

Non-TOI 

12 

3 

5 

Polarizability 

>20% 

4 

6 

4 

Percent  Error 

<20% 

82 

84 

70 

□(Easting) 

0.08 

0.09 

NA 

Position  Error 

□(Northing) 

0.09 

0.098 

NA 

□(depth) 

0.05 

0.05 

0.13 

25 


Table  7.  Classification  Results  from  Dynamic  area  of  Spencer  Range 

Due  to  an  absence  of  IMU  information  for  the  cued  TEMTADS  metrics  related  to  target  location  estimate 

accuracy  were  not  calculated. 


Dynamic  Area 

Instrument 

Metal 

Mapper  URS 

TEMTADS 

2x2 

TEMTADS 

2x2 

Metal 

Mapper 

Site 

Dynamic 

Dynamic 

Dynamic 

Dynamic 

Survey  Type 

Dynamic 

Cued 

Dynamic 

Cued 

Analyst 

Pasion 

Song 

Billings 

Zelt 

Anomaly 

Statistics 

Total  Anomalies 

339 

339 

339 

339 

Num  TOI 

23 

23 

23 

23 

Num  Non-TOI 

316 

316 

316 

316 

FAR 

Percent  of  Non-TOI 

0.165 

0.085 

0.18 

0.047 

Num  Non-Toi  Dug 

52 

27 

57 

15 

Stop  Dig  Point 

Digs 

90  (26.5%) 

67  (19.8%) 

98  (28.9%) 

58  (17.1%) 

TOI  Digs 

23  (100.0  %) 

23  (100.0  %) 

23  (100.0  %) 

23  (100.0  %) 

Non-TOI  Digs 

67  (21.2%) 

44  (13.9  %) 

75  (23.7  %) 

35  (11.1  %) 

Training  Data 

Digs 

17 

20 

22 

0 

TOI 

9 

1 

5 

0 

Non-TOI 

8 

19 

17 

0 

"Cant  Analyze" 
anomalies 

Digs 

27 

0 

0 

0 

TOI 

3 

0 

0 

0 

Non-TOI 

24 

0 

0 

0 

Polarizability 
Percent  Error 

>20% 

3 

4 

7 

3 

<20% 

18 

22 

16 

20 

Position  Error 

□(Easting) 

0.09 

NA 

0.07 

0.07 

□(Northing) 

0.06 

NA 

0.07 

0.07 

□(depth) 

0.03 

0.08 

0.07 

0.05 

7.1  PROCESSING  OPEN  AREA  DATASETS 

Cued  MetalMapper  data  were  acquired  by  URS  and  NAEVA  over  an  identical  set  of  targets 
using  the  same  instrument.  Three  different  analysts  created  ordered  dig  lists  for  each 
MetalMapper  dataset  to  ensure  a  firewall  was  maintained  for  training  data  requests  and 
groundtruth.  The  performance  of  the  various  sensors  are  summarized  in  Table  4. 

7.1.1  URS  Cued  MetalMapper  Results 

The  methodolody  for  the  URS  Cued  MetalMapper  was  described  in  detail  in  section  6. 

The  dig  list  comprised  two  stages.  For  the  early  digs  (1-201)  the  order  was  based  on 
polarizability  misfit  using  all  three  polarizabilities.  The  second  stage  (digs  202-1104)  was 
based  on  polarizability  misfit  using  only  the  primary  polarizability.  Additional  details  on 
the  classification  approach  were  provided  in  our  demonstration  report  (Pasion  et  al.,  2013). 
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The  final  receiver  operating  characteristic  (ROC)  curve  for  the  URS  MetalMapper  data  is  shown 
in  Figure  9.  At  the  stop  dig  point,  all  TOI  were  found  and  greater  than  80%  of  non-TOI  were  left 
in  the  ground. 


MM 


No  Difficult  TOIs 


Figure  9.  Final  ROC  Curve  for  Cued  MetalMapper  (URS)  for  the  Open  Area 


7.1.2  NAEVA  MetalMapper  Results 

NAEVA  MetalMapper  data  was  imported,  inverted  and  QC’d  in  a  similar  manner  as  described  in 
the  previous  section  for  the  URS  MetalMapper  data.  Training  data  selection  involved  looking 
for:  (1)  self-similar  polarizabilities;  (2)  polarizabilities  exhibiting  UXO-like  properties  distinct 
from  those  of  items  in  our  reference  library;  and  (3)  polarizabilities  that  were  similar  to  reference 
library  items  but  of  degraded  quality.  Figure  10  illustrates  a  cluster  of  self-similar  polarizabilities 
that  not  only  revealed  a  new  TOI  to  add  to  the  reference  library  (60-mm  mortar)  but  also  alerted 
the  analyst  to  the  potential  of  seeded  multi  object  scenarios  at  the  site. 

Figure  1 1  shows  the  location  in  decay-size  feature  space  of  all  training  requests.  A  single  stage 
dig  list  was  generated  where  targets  were  ranked  based  on  a  combination  of  misfit  using  all 
three  polarizabilities  (with  a  weighting  of  1)  and  decay  rate  (with  a  weighting  of  0.5).  The 
choice  of  polarizabilities  and  decay  rate  and  the  respective  weightings  was  determined 
automatically  by  allowing  the  DigZilla  software  to  search  throughout  parameter  space  using  a 
set  of  starting  weights  and  a  bisection  approach.  The  algorithm  searches  for  the  set  of  weights 
that  results  in  all  anomalies  that  have  been  flagged  as  potential/probable  TOI  being  dug  as  early 
as  possible.  A  stop  dig  point  was  determined  by  visual  inspection  of  the  predicted  polarizabilities 
(in  relation  to  the  best  fitting  reference  polarizabilities)  of  each  anomaly  plotted  in  dig  list  order. 
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The  stop  dig  point  was  conservatively  set  to  the  latest  anomaly  in  the  dig  list  with  polarizabilities 
judged  to  have  a  realistic  possibility  of  corresponding  to  a  TOI.  The  final  ROC  curve  for  the 
NAEVA  MetalMapper  data  is  shown  in  Figure  12.  Although  the  performance  of  the  non- 
automated  classification  was  slightly  better  (Figure  9),  all  TOI  were  found  and  approximately 
nearly  80%  of  non-TOI  were  left  in  the  ground  at  the  stop  dig  point. 


Polygon  1  -  Ouster  2  (7  pols) 


Figure  10.  Polarizabilities  for  a  Cluster  Identified  via  Self-similar  Polarizabilities 

Colored  lines  are  predicted  polarizabilities.  Broken  grey  lines  are  best  fitting  reference  polarizabilities. 
Training  data  were  requested  for  Anomalies  1220  and  241;  ground  truth  (photos)  revealed  that  both  of 
these  anomalies  were  60mm  mortars,  a  new  TOI.  These  items  were  added  to  the  reference  library. 
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Figure  11.  Decay  versus  Size  Feature  Space  Plot  Showing  Passed  Models  (blue  dots)  and 
the  Location  of  Training  Data  Requests  (red  dots) 

Stars  are  library  reference  items. 
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Figure  12.  Final  ROC  Curve  for  Cued  MetalMapper  (NAEVA)  for  the  Open  Area 
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7.2  PROCESSING  TREED  AREA  DATASETS 


For  this  project,  BTG  generated  a  dig  list  for  the  cued  TEMTADS  2x2  in  the  Portable  area  at 
Spencer  Range. 

7.2.1  TEMTADS  2x2  Cued  Data  Results 

Training  data  over  69  items  were  requested  and  four  TOI  were  recovered.  There  were  then  two 
stages  with  two  submitted  dig  lists  in  the  classification  process.  In  stage  one,  106  anomalies  were 
dug.  Of  these  106  anomalies,  67  were  TOIs.  No  QC  seeds  were  missed.  Based  on  the  ground 
truth  information  for  stage  one,  three  additional  anomalies  were  dug  in  stage  two.  These  three 
anomalies  were  all  non-TOI.  A  total  of  178  items  were  dug.  Five  items  were  assigned  to  the 
"Can’t  extract  reliable  parameters”  class.  As  a  result,  all  71  TOI  were  recovered.  514  anomalies 
were  left  in  the  ground  and  not  dug,  i.e.,  82.8%  of  the  non-TOI  items  were  correctly  labeled  as 
non-TOI;  all  of  the  TOI  were  correctly  identified. 

Due  to  an  absence  of  IMU  information,  metrics  related  to  target  location  estimate  accuracy  were 
not  calculated.  The  depth  estimate  errors  have  a  standard  deviation  of  13  cm,  which  exceeds  the 
success  criteria  of  less  than  10  cm.  The  survey  conditions  in  the  Treed  area  may  have  resulted  in 
variation  of  the  ground  clearance  height  of  the  instrument,  whereas  we  assumed  a  fixed  ground 
clearance  height  for  all  anomalies.  It  is  possible  that  having  a  more  accurate  measure  of  the 
ground  clearance  height  for  each  anomaly  would  reduce  the  amount  of  error  in  the  depth 
estimates. 
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Figure  13.  ROC  Curve  for  the  TEMTADS  2x2  Cued  Data  Analysis  in  the  Treed  Area 
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7.3  PROCESSING  DATA  COLLECTED  IN  THE  DYNAMIC  AREA 


In  the  Dynamic  area  at  Spencer  Range,  BTG  generated  dig  lists  for  the  cued  TEMTADS  2x2, 
dynamic  TEMTADS  2x2,  cued  URS  MetalMapper  and  dynamic  URS  MetalMapper  data.  We 
note  that  BTG  also  produced  a  dig  list  for  the  cued  and  dynamically  acquired  MPV  data  as  part 
of  ESTCP  MR-201158.  We  refer  the  reader  to  the  ESTCP  MR-201158  Spencer  Demonstration 
report  for  MPV  processing  details  and  results. 

7.3.1  URS  Cued  MetalMapper  Results 

Cued  URS  MetalMapper  data  in  the  Dynamic  area  were  analyzed  by  the  same  BTG  analyst  who 
previously  generated  a  cued  dig  list  for  the  URS  open  area  MetalMapper  data.  A  similar 
approach  to  the  one  used  for  the  open  area  was  applied  to  the  dynamic  area  data.  No  training  data 
were  requested  for  the  dynamic  area  as  it  was  assumed  that  the  ground  truth  information  that  had 
been  received  (up  to  the  stop  dig  point)  for  the  open  area  was  sufficient  training  data.  During 
visual  QC  of  the  data,  a  new  class  of  suspicious,  but  unknown,  TOI  was  added  to  the  ordnance 
library  based  on  the  polarizabilities  for  Anomalies  SR-1729  and  SR-1550.  Both  of  these  turned 
out  to  be  37 -mm  TOI.  One  of  these  (SR-1729)  was  a  multi-object  scenario  (i.e.,  37-mm  with  four 
medium  to  large  pieces  of  frag).  The  training  data  tool  was  used  to  look  for  other  “hidden” 
clusters,  but  none  were  found. 


The  classification  approach  was  similar  to  that  used  for  the  open  area  data.  The  dig  list 
comprised  two  stages.  For  the  early  digs  (1-39),  the  order  was  based  on  polarizability  misfit 
using  all  three  polarizabilities.  The  second  stage  (digs  40-339)  was  based  on  polarizability  misfit 
using  only  the  primary  polarizability.  The  stop  dig  point  was  set  at  dig  number  55. 


Figure  14.  Final  ROC  Curve  for  Cued  MetalMapper  (URS)  for  the  Dynamic  Area 
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7.3.2  URS  Dynamic  MetalMapper  Results 


MetalMapper  data  acquired  in  a  dynamic,  full  coverage  mode  were  processed  using  UXOLab. 
The  data  were  background  corrected  by  identifying  regions  in  the  data  that  did  not  contain  data 
anomalies,  and  then  using  these  anomaly-free  regions  to  estimate  background  levels  which 
would  subsequently  be  removed  from  the  data. 

BTG/UBC-GIF  did  not  carry  out  any  target  picking  or  anomaly  selection.  An  anomaly  list  was 
provided  by  the  ESTCP  program  office.  Data  within  a  1.5  m2  region  surrounding  each  anomaly 
flag  was  extracted  from  the  background  corrected  dataset,  and  a  one  and  two  dipole  source 
inversion  was  used  to  provide  location,  orientation  and  dipole  polarizability  estimates. 

We  performed  a  semi-automated  cluster  analysis  to  search  for  clusters  of  items  with  self-similar 
polarizabilities.  Clusters  were  formed  in  three  ways:  (1)  total  polarizability;  (2)  primary 
polarizability;  and  (3)  1st  and  2nd  largest  polarizabilities.  Clusters  that  were  judged  to  be  UXO- 
like  or  vaguely-UXO  like  were  selected  as  clusters  of  interest.  From  these  clusters  of  interest,  we 
typically  selected  a  couple  of  example  items  for  training  data.  In  addition,  anomalies  lying  on  the 
edge  of  clusters  of  very  likely  UXO-like  items  would  be  selected  for  training  data  as  a  means  of 
gauging  both  the  potential  variability  of  a  cluster  class,  and  the  quality  of  the  data.  A  single 
training  data  request  of  17  anomalies  was  submitted  to  the  program  office. 

The  CCR  algorithm  was  used  for  creating  a  prioritized  dig  list.  Dig  list  order  is  based  on  four 
metrics:  (1)  best  polarizability  misfit  relative  to  a  library  of  reference  ordnance  items  calculated 
using  all  three  polarizabilities  (Li,  L2  and  Lj);  (2)  same  as  (1)  but  calculated  using  only  the 
primary  (Li)  polarizability;  (3)  polarizability  size;  and  (4)  polarizability  decay.  The  anomalies 
are  sorted  according  to  each  metric,  creating  four  lists  of  ordinals  for  each  anomaly.  The  final 
“score”  for  each  anomaly  is  a  weighted  sum  of  the  ordinals. 

The  ROC  curve  for  the  submitted  list  is  shown  in  Figure  15.  At  the  stop-dig  point  -  excluding 
training  data  digs  and  “can’t  analyze”  digs  -  there  were  90  total  digs,  of  which  23  were  TOI  and 
67  were  non-TOI.  At  this  operating  point,  100%  of  the  TOI  were  identified  for  excavation  and 
21.2%  of  the  non-TOI  anomalies  were  marked  for  excavation.  The  FAR  was  0.165,  with  52  of 
the  316  non-TOI  excavated. 

All  performance  metrics  related  to  the  accuracy  of  estimated  target  location  were  successfully 
met  (Table  5).  A  percent  error  was  calculated  to  quantify  the  misfit  between  estimated 
polarizabilities  and  the  library  polarizability  corresponding  to  the  ground  truth.  The  percent  error 
is  calculated  using  all  three  principal  polarizabilities.  Only  three  TOI  exceeded  the  20% 
threshold  for  the  polarizability  percent  error  metric.  Of  these  three  TOI,  SR- 1502  was  identified 
in  the  training  data  stage  of  the  classification,  and  the  other  two  TOI  (SR- 1506  and  SR- 1729)  had 
a  low  enough  primary  polarizability  misfit  to  be  selected  for  excavation. 
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Figure  15.  Final  ROC  Curve  for  MetalMapper  (URS)  Data  Acquired  in  Dynamic  Mode 
7.3.3  Cued  TEMTADS  2x2  Results 

The  analyst  that  processed  TEMTADS  2x2  cued  data  in  the  Treed  area  also  processed 
TEMTADS  2x2  data  in  the  Dynamic  area.  The  polarizability  library  used  in  the  Treed  area  was 
used  for  the  dynamic  area.  The  process  for  requesting  training  data  was  similar  to  that  used  in  the 
Treed  area,  and  resulted  in  a  total  of  20  training  anomalies. 

A  single  classification  dig  list  was  submitted.  No  QC  seeds  were  missed  in  this  initial  dig  list. 
After  reviewing  the  ground  truth  information  for  these  anomalies,  it  was  determined  that  it  was 
unlikely  that  there  would  be  additional  TOI  in  the  remaining  anomalies.  All  23  TOI  were 
correctly  marked  for  excavation.  86.9%  of  the  non-TOI  items  were  correctly  labeled  as  non-TOI 
while  retaining  all  of  the  TOI  on  the  dig  list.  Figure  16  shows  the  final  ROC  curve  provided  by 
IDA. 

All  performance  metrics  related  to  classification  of  TOI  and  non-TOI  satisfied  the  success 
criteria.  Location  estimates  were  not  calculated,  due  to  the  absence  of  IMU  information  available 
for  the  survey.  The  standard  deviation  of  the  depth  estimate  error  is  0.08  m.  A  percent  error  was 
calculated  to  quantify  the  misfit  between  estimated  polarizabilities  and  the  library  polarizability 
corresponding  to  the  ground  truth.  The  percent  error  is  calculated  using  all  three  principal 
polarizabilities.  Only  4  TOI  exceeded  the  20%  threshold  for  the  polarizability  percent  error 
metric.  Of  these  4  TOI,  SR- 1502  was  identified  in  the  training  data  stage  of  the  classification. 
Two  of  the  4  TOI  (SR-1564  and  SR-1576)  had  data  recollected,  for  which  the  polarizability  fit  of 
the  recollected  data  had  a  percent  error  that  met  the  success  criteria.  Although  anomaly  SR- 1609 
had  a  poor  fit  to  the  library  item,  it  was  still  dug  just  before  the  stop-dig  point. 
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Figure  16.  Final  ROC  Curve  for  TEMTADS  2x2  Cued  Data  Acquired  in  the  Dynamic 

Area 


7.3.4  Dynamic  TEMTADS  2x2  Results 

The  background  corrected  TEMTADS  2x2  dynamic  data  collected  at  Spencer  Range  were 
inverted  using  a  sequential  inversion  approach  to  estimate  target  location,  depth  and  principal 
polarizabilities.  Two  SOIs  were  performed  per  anomaly:  (1)  using  all  data  within  75  cm  of  the 
picked  target  location;  and  (2)  by  inverting  only  the  data  along  the  transect  that  passed  closest  to 
the  picked  target  location.  Where  necessary,  the  analyst  adjusted  masks  and/or  fit  a  201  model  to 
the  data.  The  best  fitting  model  for  any  of  the  inversions  performed  for  a  particular  anomaly  was 
used  to  make  classification  decisions. 

Analysis  of  the  data,  including  visual  QC  of  data  and  model  parameters,  selection  of  training  data, 
and  dig  list  creation,  was  performed  using  the  UXOLab  software  suite.  Visual  QC  of  the  data  was 
performed  using  QCZilla,  which  allowed  us  to  compare  the  observed  and  predicted  data,  review 
predicted  model  parameters,  and  examine  measures  of  data/model  quality.  Predicted  polarizabilities 
were  compared  to  reference  polarizabilities  for  various  ordnance  items  initially  derived  from  IVS 
and  test  pit  measurements.  As  the  analysis  proceeded,  the  library  of  reference  items  was  augmented 
with  additional  items  based  on  ground  truth  obtained  through  training  data  requests. 

Training  data  information  was  requested  for  twenty-one  items  spread  across  two  requests.  The 
training  data  requests  focused  on  items  with  similar  size  and  time-decay  parameters  to  the  known 
ordnance  items  and  items  that  appeared  to  exhibit  axial  symmetry.  Figure  17  shows  the  location 
in  decay-size  feature  space  of  all  training  requests. 
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Figure  17.  Decay  Versus  Size  Feature  Space  Plot  Showing  Passed  Models  (blue  dots) 
and  the  Location  of  Training  Data  Requests  (red  dots) 

Stars  are  library  reference  items. 

A  single  stage  dig  list  was  generated  in  which  targets  were  ranked  using  the  CCR  algorithm  using 
feature  vectors  comprising:  (1)  all  polarizabilities;  (2)  primary  polarizability;  (3)  size;  and  (4)  decay. 
Rankings  for  each  of  the  four  sets  of  feature  vectors  are  obtained  by  comparison  to  the  equivalent 
features  in  the  reference  library.  The  CCR  is  obtained  by  the  weighted  sum  of  the  rankings  in  the 
four  separate  ranking  schemes.  Thus,  a  feature  vector  that  ranks  high  in  more  than  one  scheme  will 
rank  high  in  the  CCR.  The  stop  dig-point  was  determined  subjectively  by  the  analyst;  digging 
ceased  after  all  items  deemed  to  be  high-priority  TOI  and/or  low-confidence  non-TOI  items  were 
dug.  In  the  original  dig-list  submission,  one  seed  item  (SR- 1676)  occurred  past  the  stop-digging 
point.  This  item  had  a  lower  than  expected  time-decay  parameter  that  placed  it  just  past  the  stop-dig 
point.  The  CCR  weights  were  adjusted  so  that  items  in  the  region  of  feature  space  below  the  ISO 
and  37-mm  feature  vectors  were  ranked  higher.  The  final  ROC  curve  for  the  dynamically  acquired 
TEMTADS  2x2  data  in  the  dynamic  area  is  shown  in  Figure  18.  All  TOI  were  identified  for 
excavation  and  greater  than  80%  of  non-TOI  were  left  in  the  ground  at  the  stop-dig  point. 

7.4  TECHNOLOGY  TRAINING  AND  TRANSFER 

A  member  of  the  Shaw  Environmental  production  team  attended  a  one  week  training  session  in 
Vancouver,  B.C.,  Canada  with  BTG  algorithm  and  software  developers.  The  training  session 
included  an  overview  of  UXO  inversion  and  classification  theory  and  software  routines.  The 
Shaw  geophysicist  was  responsible  for  executing  all  parts  of  the  classification  workflow:  from 
data  and  inversion  QC,  training  data  selection,  to  diglist  creation  and  submittal.  The  final  ROC 
curve  is  shown  in  Figure  19. 


35 


Percent  of  TOIs  Correctly  Classified  (%)  53  Percent  of  TOIs  Correctly  Classified  (%) 


2x2 


No  Difficult  TOIs 


18.  Final  ROC  Curve  for  the  Dynamic  TEMTADS  2x2  Data  from  the  Dynamic 

Survey 


MM 
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Figure  19.  ROC  Curve  for  the  Library  Based  Classification  of  MetalMapper  (URS) 
Data  Carried  Out  by  a  Shaw  Environmental  Geophysicist 
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8.0  COST  ASSESSMENT 


The  majority  of  the  work  reported  here  was  performed  while  the  data  analysts  were  employed  by 
Sky  Research.  When  the  project  team  moved  to  BTG,  they  lost  access  to  the  labor  information 
required  to  meaningfully  compare  the  costs  of  the  different  technologies.  Here,  we  reproduce  the 
cost  analysis  provided  by  the  Program  Office  in  the  ESTCP  (2014)  report  on  the  Spencer 
demonstration,  which  was  specifically  for  the  MetalMapper  deployed  in  dynamic  mode. 

8.1  COST  MODEL 

The  demonstration  took  place  on  a  small  part  of  the  former  Spencer  Artillery  Range  and  incurred 
costs  for  many  items  specific  to  a  demonstration  that  would  not  be  needed  in  an  application  of 
classification  to  a  real  site.  Nevertheless,  we  can  extract  meaningful  projected  performance  for 
the  technology  and  apply  reasonable  industry  unit  costs  for  various  elements  to  arrive  at  a  total 
cost  comparison  for  clearing  an  example  100-acre  site  with  and  without  the  use  of  classification. 

We  made  the  following  assumptions: 

•  The  example  takes  place  in  an  area  with  similar  munitions  types  and  the  same  density  of 
anomalies  as  seen  in  this  demonstration.  There  were  approximately  250  anomalies  per 
acre  at  the  demonstration  site.  Extrapolating,  we  would  expect  about  25,000  anomalies  in 
a  similar  100-acre  area,  with  24,925  clutter. 

•  Mob/demob  and  a  surface  sweep  would  be  required  for  either  scenario. 

•  Three  TOI  were  found  during  the  intrusive  investigation  of  the  ~4-acre  area.  We  predict 
about  75  native  TOI  in  a  100-acre  site. 

•  The  baseline  is  an  EM-61  survey  with  0.5  m  line  spacing.  This  would  be  used  to  select 
anomalies  for  digging  without  classification  and  for  cueing  with  the  advanced  sensor  in 
hybrid  mode. 

•  Survey  data  will  be  collected  using  MetalMapper  in  dynamic  mode  for  the  advanced 
sensor  only  mode. 

•  The  site  is  seeded  at  a  rate  so  on  average  one  seed  will  be  encountered  each  day  of 
MetalMapper  data  collection.  With  an  estimate  of  25,000  total  anomalies  and  a 
production  rate  of  200  anomalies  per  day,  we  seed  125  inert  items.  These  QC  seeds 
would  be  used  whether  classification  was  used  on  the  site  or  not. 

•  The  classification  performance  is  as  achieved  by  the  production  firms  with  -80%  of  the 
clutter  correctly  identified  and  remaining  undug. 

•  The  unit  costs  are  as  shown  in  Table  8. 

With  these  assumptions,  the  costs  were  calculated  using  the  elements  shown  in  Table  9.  A  49% 
savings  on  project  field  work  can  be  realized  using  a  hybrid  approach,  in  which  anomalies  are 
identified  from  an  EM-61  -MK2  survey  and  cued  data  is  collected  with  the  MetalMapper  (the 
method  used  in  this  demonstration). 
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Table  8. 


Unit  Cost  Assumptions 


Item 

Units 

Cost 

Mob/Demob 

1 

$7,000 

Surface  Sweep 

acre 

$2,500 

IVS  and  Seed  Emplacement 

125  seeds 

$22,650 

EM-61  Survey  Data  Collection  and  Analysis 

acre 

$1,000 

Dynamic  MetalMapper  Data  Collection  and  Analysis 

acre 

$5,000 

Cued  MetalMapper  Data  Collection  and  Analysis 

per  anomaly 

$30 

Digs 

per  dig 

$125 

Table  9.  Cost  Comparison  for  100  Acres  of  Comparable  Spencer  Range  Site 


No  Classification 

Hybrid  Approach1 

Advanced 

Sensor 

Only2 

Work  Item 

Quantity 

Cost/$ 

Quantity 

Cost/$ 

Quantity 

Cost/$ 

Mob/demob 

i 

7,000 

i 

7,000 

i 

7,000 

Surface  Sweep 

100  acres 

250,000 

100  acres 

250,000 

100  acres 

250,000 

Seeds 

125  items 

22,650 

125  items 

22,650 

125  items 

22,650 

EM-61  Survey 

100  acres 

100,000 

100  acres 

100,000 

n/a 

MetalMapper 

Survey 

n/a 

n/a 

100  acres 

500,000 

Cued 

MetalMapper 

n/a 

25,000 

anomalies 

750,000 

8,333 

anomalies 

249,990 

Seeds  Dug 

125 

15,625 

125 

15,625 

125 

15,625 

Native  UXO  Dug 

75 

9,375 

75 

9,375 

75 

9,375 

Clutter  Dug 

24,925 

3,115,625 

4,985 

623,125 

4,985 

623,125 

TOTAL 

3,520,275 

1,777,775 

1,677,765 

Percent  Savings 

49% 

52% 

1  Hybrid  approach  denotes  the  method  used  in  this  demonstration,  anomaly  detection  from  an  EM-61-MK2 
survey  with  cued  data  collection  using  advanced  sensors. 


2  Advanced  sensor  only  approach  refers  to  anomaly  detection  and  classification  of  2/3  of  the  anomalies  using 
MetalMapper  in  dynamic  mode  and  cued  data  collection  on  the  remaining  1/3  of  the  anomalies. 

8.2  COST  BENEFIT 

As  the  Classification  demonstration  program  has  progressed,  it  has  become  apparent  that  a 
significant  number  of  anomalies  can  be  classified  using  data  collected  dynamically  using 
advanced  EMI  sensors.  This  raises  the  possibility  of  performing  the  anomaly  detection  step  using 
advanced  sensors,  classifying  many  of  the  anomalies  using  the  same  data  set  (2/3  of  the 
anomalies  in  a  recent  test),  and  only  having  to  collect  cued  data  over  a  subset  of  the  anomalies. 
The  cost  estimate  for  this  “Advanced  Sensor  Only”  mode  is  given  in  the  rightmost  two  columns 
in  Table  9. 
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Given  the  reduced  number  of  data  collections,  one  might  expect  the  savings  from  the  advanced 
sensor  only  mode  to  be  much  greater  than  the  hybrid  approach.  The  current  generation  of 
advanced  sensors  are  not  configured  to  be  deployed  in  arrays,  so  the  cost  for  the  advanced  sensor 
detection  survey  for  this  example  site  is  much  higher  than  for  detection  using  an  EM-61-MK2 
array.  As  the  operation  of  these  advanced  sensors  in  dynamic  mode  is  explored  further,  it  is 
possible  that  this  cost  disadvantage  will  lessen. 
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9.0  IMPLEMENTATION  ISSUES 


Specific  implementation  issues  related  to  the  technologies  demonstrated  here  are: 

•  Access  to  good,  high-quality,  reliable  hardware  for  collection  of  data  suitable  for 
Advanced  Geophysical  Classification  (AGC)  continues  to  be  a  problem  for  the  industry. 

•  UXOLab  software  was  used  for  all  the  processing  and  interpretation  activities  described 
in  the  report.  While  UXOLab  is  available  under  license  from  UBC,  it  is  not  suitable  for 
general  distribution  to  government  contractors.  Firstly,  using  the  software  successfully 
requires  advanced  knowledge  of  geophysical  inversion  and  statistical  classification. 
Secondly,  while  the  software  doesn’t  require  the  user  to  have  a  Matlab  license,  it  was 
built  entirely  within  the  Matlab  software  environment  to  support  the  needs  of  UXO 
researchers.  Thirdly,  UBC  is  not  set  up  to  provide  maintenance  and  support  for  the 
software.  As  part  of  Strategic  Environmental  Research  and  Development  Program 
(SERDP)  MR-2318,  BTG  are  in  the  process  of  porting  the  Matlab  code  to  C++  so  that  the 
software  will  be  accessible  with  Geosoft  Oasis  Montaj  (the  industry  standard  processing 
software). 

•  Access  to  the  appropriate  software  and  hardware  is  the  first  critical  step  in  successful 
implementation  of  the  technology.  Ensuring  that  operators  have  the  appropriate  level  of 
training  and  experience  is  the  second.  This  requirement  for  training  and  experience 
applies  to  both  the  feature  vector  extraction  and  classification  steps.  While  the  process  of 
feature  extraction  is  straightforward  and  reliable  for  a  large  number  of  anomalies,  there  is 
always  a  certain  percentage  of  anomalies  where  the  default  parameters  don’t  produce  a 
sensible  or  reliable  result.  Recognizing  those  situations  is  critical  to  minimize  the 
probability  of  a  false  negative,  while  being  able  to  appropriately  adjust  the  strategy  is 
important  for  reducing  the  FP  count.  For  classification,  each  site  has  unique  challenges 
and  the  most  appropriate  classification  strategy  changes.  Thus,  each  analyst  must  be 
capable  of  exploring  (and  understanding  the  potential  pitfalls  of)  different  feature  spaces 
that  could  be  used  for  discrimination.  In  summary,  we  believe  that  a  very  solid 
understanding  of  both  feature  extraction  and  classification  are  required  for  the 
discrimination  technology  to  be  reliably  applied  to  different  DoD  sites. 
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APPENDIX  A  POINTS  OF  CONTACT 


Points  of  contact  (POCs)  involved  in  the  demonstration  and  their  contact  information  are 
presented  below. 


Point  of  Contact 
Name 

Organization 

Name 

Address 

Phone 

Fax 

Email 

Role  in  Project 

Dr.  Leonard  Pasion 

Black  Tusk  Geophysics,  Inc. 
401  /1755  West  Broadway, 
Vancouver,  BC 

VGJ  4S5,  Canada 

Tel:  604-428-3382 

leonard.pasion@btseophvsics.com 

Principal 

Investigator 

Kevin  Kingdon 

Black  Tusk  Geophysics,  Inc. 
Same  address 

Tel:  604  428  3382 
kevin.kingdon@btgeophysics.com 

Project 

management  and 
personnel 
coordination 

Dr.  Barry  Zelt 

Black  Tusk  Geophysics,  Inc. 
Same  address 

Tel:  604  428  3382 
barry.zelt@btgeophysics.com 

Data  analyst  and 
UXOLab 
programming 

Dr.  Stephen 
Billings 

Black  Tusk  Geophysics,  Inc. 
Same  address 

Tel:  +1  720  306  1165 
stephen.billings@btgeophysics.com 

Data  analyst 

Dr.  Laurens  Beran 

Black  Tusk  Geophysics,  Inc. 
Same  address 

604  428  3382 

laurens  .beran  @  btgeophy  sics .  com 

Data  analyst, 
classification 
theory 

Dr.  Herb  Nelson 

ESTCP  Program  Office 

4800  Mark  Center  Drive 

Suite  17D08 

Alexandria,  VA  22350-3605 

Tel:  571-372-6400 

Herbert.Nelson@osd.mil 

ESTCP  Munitions 
Management 
Program  Manager 
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